ptest-cargo: fix quoted RUSTFLAGS passed to cargo
PATCH 1/1 ptest-cargo: fix quoted RUSTFLAGS passed to cargo
From: AnnLeelu George <[email protected]>
do_compile_ptest_cargo passes RUSTFLAGS directly to the cargo environment, bypassing the shell. As a result, BitBake's shell escaping is preserved and can cause quoted cfg values to reach rustc incorrectly.
This can be reproduced with:
DISTRO_FEATURES:append = " ptest"
DEPRECATED_CRYPTO_FLAGS:pn-openssl = "no-des"
OpenSSL then defines OPENSSL_NO_DES, which openssl-sys emits as an osslconf cfg. The resulting RUSTFLAGS contains:
--cfg osslconf=\"OPENSSL_NO_DES\"
The rpm-sequoia ptest build consequently fails with:
error: unknown start of token error[E0765]: unterminated double quote string
The normal cargo.bbclass path is unaffected because RUSTFLAGS is exported through the shell, which removes the shell escaping before cargo sees it.
Parse RUSTFLAGS with shlex.split() before passing it to cargo so that the value is normalized in the same way as the shell-based cargo path.
--- meta/classes-recipe/ptest-cargo.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes-recipe/ptest-cargo.bbclass +2 -1
@@ -11,6 +11,7 @@ CARGO_TEST_BINARIES_FILES ?= "${B}/test_binaries_list" python do_compile_ptest_cargo() { import subprocess import json + import shlex cargo = bb.utils.which(d.getVar("PATH"), d.getVar("CARGO")) cargo_build_flags = d.getVar("CARGO_BUILD_FLAGS")
@@ -21,7 +22,7 @@ python do_compile_ptest_cargo() { manifest_dir = os.path.dirname(manifest_path) env = os.environ.copy() - env['RUSTFLAGS'] = rust_flags + env['RUSTFLAGS'] = ' '.join(shlex.split(rust_flags)) cmd = f"{cargo} build --tests --message-format json {cargo_build_flags} {packageconfig_confargs}" bb.note(f"Building tests with cargo ({cmd})")
Signature · 1 line
2.49.0
Quoted 15 lines · Deepesh via lists.openembedded.org
From: AnnLeelu George <[email protected]>
do_compile_ptest_cargopasses RUSTFLAGS directly to the cargo environment, bypassing the shell. As a result, BitBake's shell escaping is preserved and can cause quoted cfg values to reach rustc incorrectly.This can be reproduced with:
DISTRO_FEATURES:append = " ptest"
DEPRECATED_CRYPTO_FLAGS:pn-openssl = "no-des"OpenSSL then defines
OPENSSL_NO_DES, which openssl-sys emits as an osslconf cfg. The resulting RUSTFLAGS contains:
Quoted 6 lines
--cfg osslconf=\"OPENSSL_NO_DES\"The rpm-sequoia ptest build consequently fails with:
error: unknown start of token
error[E0765]: unterminated double quote string
The need to shlex.split a variable that is set by the recipe seemed unusual to me, so I poked a little into this but I can’t replicate this behaviour. rpm-sequotia does indeed fail to build with the reproduction steps, but printing the value of RUSTFLAGS at the point where this patch uses shlex gives the following string:
WARNING: rpm-sequoia-1.10.2-r0 do_compile_ptest_cargo: rust_flags=' -L /work/ross/build/tmp/work/cortexa57-poky-linux/rpm-sequoia/1.10.2/recipe-sysroot/usr/lib/rustlib/aarch64-poky-linux-gnu/lib --remap-path-prefix=/work/ross/build/tmp/work/cortexa57-poky-linux/rpm-sequoia/1.10.2=/usr/src/debug/rpm-sequoia/1.10.2 --remap-path-prefix=/work/ross/build/tmp/work-shared=/usr/src/debug/rpm-sequoia/1.10.2 ‘
No quotation marks in there. The build does then fail, but (after undoing several layers of json and escaping) the error is:
aarch64-poky-linux/aarch64-poky-linux-ld:
libopenssl-3aad21d8f08654a9.rlib(openssl-3aad21d8f08654a9.openssl.4278bd4403a9a010-cgu.10.rcgu.o):
in function `<openssl::cipher::Cipher>::des_ede3_cbc':
openssl.4278bd4403a9a010-cgu.10:(.text._RNvMs0_NtCs5HPgRTJKBnE_7openssl6cipherNtB5_6Cipher12des_ede3_cbc+0x0):
undefined reference to `EVP_des_ede3_cbc'
aarch64-poky-linux/aarch64-poky-linux-ld:
libopenssl-3aad21d8f08654a9.rlib(openssl-3aad21d8f08654a9.openssl.4278bd4403a9a010-cgu.10.rcgu.o):
in function `<openssl::cipher::Cipher>::des_ede3_ecb':
openssl.4278bd4403a9a010-cgu.10:(.text._RNvMs0_NtCs5HPgRTJKBnE_7openssl6cipherNtB5_6Cipher12des_ede3_ecb+0x0):
undefined reference to `EVP_des_ede3_ecb'
aarch64-poky-linux/aarch64-poky-linux-ld:
libopenssl-3aad21d8f08654a9.rlib(openssl-3aad21d8f08654a9.openssl.4278bd4403a9a010-cgu.10.rcgu.o):
in function `<openssl::cipher::Cipher>::des_ede3_cfb64':
openssl.4278bd4403a9a010-cgu.10:(.text._RNvMs0_NtCs5HPgRTJKBnE_7openssl6cipherNtB5_6Cipher14des_ede3_cfb64+0x0):
undefined reference to `EVP_des_ede3_cfb64'
Which seems fair, considering we just disabled the des functions.
Can you reproduce using plain oe-core? I wonder if the problem you see is specific to WR Linux?
Thanks,
Ross
Quoted 50 lines · Ross Burton
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 14 lines · Deepesh via lists.openembedded.org
From: AnnLeelu George<[email protected]>
do_compile_ptest_cargopasses RUSTFLAGS directly to the cargo environment, bypassing the shell. As a result, BitBake's shell escaping is preserved and can cause quoted cfg values to reach rustc incorrectly.This can be reproduced with:
DISTRO_FEATURES:append = " ptest"
DEPRECATED_CRYPTO_FLAGS:pn-openssl = "no-des"OpenSSL then defines
OPENSSL_NO_DES, which openssl-sys emits as an osslconf cfg. The resulting RUSTFLAGS contains:Quoted 6 lines
--cfg osslconf=\"OPENSSL_NO_DES\"The rpm-sequoia ptest build consequently fails with:
error: unknown start of token error[E0765]: unterminated double quote string
The need to shlex.split a variable that is set by the recipe seemed unusual to me, so I poked a little into this but I can’t replicate this behaviour. rpm-sequotia does indeed fail to build with the reproduction steps, but printing the value of RUSTFLAGS at the point where this patch uses shlex gives the following string:
WARNING: rpm-sequoia-1.10.2-r0
do_compile_ptest_cargo: rust_flags=' -L /work/ross/build/tmp/work/cortexa57-poky-linux/rpm-sequoia/1.10.2/recipe-sysroot/usr/lib/rustlib/aarch64-poky-linux-gnu/lib --remap-path-prefix=/work/ross/build/tmp/work/cortexa57-poky-linux/rpm-sequoia/1.10.2=/usr/src/debug/rpm-sequoia/1.10.2 --remap-path-prefix=/work/ross/build/tmp/work-shared=/usr/src/debug/rpm-sequoia/1.10.2 ‘No quotation marks in there. The build does then fail, but (after undoing several layers of json and escaping) the error is:
aarch64-poky-linux/aarch64-poky-linux-ld:
libopenssl-3aad21d8f08654a9.rlib(openssl-3aad21d8f08654a9.openssl.4278bd4403a9a010-cgu.10.rcgu.o):
in function `<openssl::cipher::Cipher>::des_ede3_cbc':
openssl.4278bd4403a9a010-cgu.10:(.text._RNvMs0_NtCs5HPgRTJKBnE_7openssl6cipherNtB5_6Cipher12des_ede3_cbc+0x0):
undefined reference to `EVP_des_ede3_cbc'aarch64-poky-linux/aarch64-poky-linux-ld:
libopenssl-3aad21d8f08654a9.rlib(openssl-3aad21d8f08654a9.openssl.4278bd4403a9a010-cgu.10.rcgu.o):
in function `<openssl::cipher::Cipher>::des_ede3_ecb':openssl.4278bd4403a9a010-cgu.10:(.text._RNvMs0_NtCs5HPgRTJKBnE_7openssl6cipherNtB5_6Cipher12des_ede3_ecb+0x0):
undefined reference to `EVP_des_ede3_ecb'aarch64-poky-linux/aarch64-poky-linux-ld:
libopenssl-3aad21d8f08654a9.rlib(openssl-3aad21d8f08654a9.openssl.4278bd4403a9a010-cgu.10.rcgu.o):
in function `<openssl::cipher::Cipher>::des_ede3_cfb64':openssl.4278bd4403a9a010-cgu.10:(.text._RNvMs0_NtCs5HPgRTJKBnE_7openssl6cipherNtB5_6Cipher14des_ede3_cfb64+0x0):
undefined reference to `EVP_des_ede3_cfb64'
Quoted 6 lines
Which seems fair, considering we just disabled the des functions.
Can you reproduce using plain oe-core? I wonder if the problem you see is specific to WR Linux?
Thanks,
Ross
Hi Ross,
Thanks for reviewing this. As you pointed out, *the error[E0765]: unterminated double quote string* issue is specific to WR Linux.
However, the linker errors we observe in oe-core when setting DEPRECATED_CRYPTO_FLAGS:pn-openssl = "no-des" indicate a potential issue that needs further investigation ?
To address this, we submitted and backported the upstream Sequoia commit https://gitlab.com/sequoia-pgp/sequoia/-/commit/fcc286417cdf9152e7df1a6152d31388a0ca9d24 , Fix this issue by checking OPENSSL_NO_DES before using the related functions in sequoia-openpgp and we pass --cfg osslconf=\"OPENSSL_NO_DES\" to RUSTFLAGS by checking DEPRECATED_CRYPTO_FLAGS has no-des or not.
This raises two questions regarding oe-core:
Is this also a potential issue in plain oe-core?
If rpm-sequoia is built against an OpenSSL configuration with DES disabled, would oe-core also need to backport the upstream change and explicitly pass --cfg osslconf="OPENSSL_NO_DES" to rustc?
Could you confirm whether this configuration needs to be handled in oe-core ?
Can the RUSTFLAGS quoting issue be fixed without using shlex.split()?
When building with ptest enabled, we encounter:
/ error: unknown start of token error[E0765]: unterminated double quote string/
The underlying issue appears to be that cargo_do_compile and do_compile_ptest_cargo handle RUSTFLAGS differently. The cargo_do_compile exports it through the shell, whereas the do_compile_ptest_cargo sets the environment variable directly from Python. Consequently, the quoting required for the cfg argument is not handled consistently.
We are considering using CARGO_ENCODED_RUSTFLAGS instead:
/python () {
if bb.utils.contains('DEPRECATED_CRYPTO_FLAGS', 'no-des', True, False, d):
flags = (d.getVar('RUSTFLAGS') or '').split() + [ '--cfg', 'osslconf="OPENSSL_NO_DES"'
]
d.setVar('CARGO_ENCODED_RUSTFLAGS', '\x1f'.join(flags)) d.setVarFlag('CARGO_ENCODED_RUSTFLAGS', 'export', '1')
}/
Cargo documents CARGO_ENCODED_RUSTFLAGS <https://doc.rust-lang.org/cargo/reference/environment-variables.html#:~:text=see%20CARGO_ENCODED_RUSTFLAGS.-,CARGO_ENCODED_RUSTFLAGS,-%E2%80%94%20A%20list%20of> as accepting compiler arguments separated by the ASCII unit separator (\x1f), avoiding shell-dependent quoting. This should allow the same arguments to be passed through both compile paths.
Would this approach be preferable to using shlex.split() to work around the quoting issue? Alternatively, is there a more appropriate way to handle this in the existing Cargo classes?
I'd appreciate your thoughts on whether any corresponding change is needed in oe-core, and on the preferred way to handle the flags consistently across both Cargo task paths.
Regards,
Deepesh